Regarding pdpyras - I would like some input

#1

Hello PagerDuty community!

I wanted to propose an idea for an update ahead of pdpyras v4 that would actually help guide development of v4. An improvement I’m hoping to add to the library in v4, time permitting, is a new pagination method that is more resistant to race conditions and can be safely used for updating/deleting records en masse.

Would anyone strongly object to setting a custom user-agent HTTP header that identifies the client as pdpyras, including its version number and the Python version?

The reason I ask is because it would make it possible for PagerDuty to identify usage of this API client library, and thus in a certain sense, gather usage information that we previously did not gather.

For a more in-depth description of how and under what circumstances pagination can be affected by concurrent create/update/delete operations, see Issue #9 and also Disclaimers Regarding Iteration in the readme on the dev branch.

Update: the new version has been released! Versions 3.1 and later should set a distinct User-Agent header in all requests.

#2

I’m actually surprised this isn’t already being done with go-pagerduty or PDJS. :open_mouth:

Is the idea just that more PagerDuty resources can be allocated to maintain pdpyras (and ensure compatibility with the most commonly used versions of Python) with some hard usage data?

1 Like
#3

Yes, indeed; I definitely would like to better gauge how widely used it is (and also what versions of Python I should prioritize for testing/support). The primary goal however was to get some sense of the average sizes of datasets involved when iterating through index endpoints.

The reason for this is because the larger the number of results, the longer it takes to fetch all of them, and thus the higher the likelihood that some other operation that potentially affects iteration will end up taking place at the same time (i.e. users editing objects in the web UI).

So, in effect, I’m trying to gauge how much need there is for devising some clever improvement to iter_all, i.e. keep track of total and grope around / pick a new offset parameter to use if total changes mid-iteration, so that no records are skipped and the number of GET requests is still kept to a minimum.

If we can make the assumption that the set will either not change mid-iteration, or change by exactly the page size (i.e. because the update/delete operation removed all the results), the design could be kept a lot simpler – i.e. keep the offset parameter zero in a hypothetical delete_all method instead of incrementing it, or even more simply, just pre-fetch all results in any case.

What spurred this was a few bugs in our internal tools that had the exact same cause: deleting or updating things mid-iteration versus pre-fetching and then operating over the whole list, which caused some results to get skipped.

2 Likes
#4

I also agree that adding a user-agent header to pdpyras and the other libraries is an awesome idea!

1 Like
#5

I’d absolutely support a custom user-agent. That’d be a great way to highlight usage. :slight_smile:

I’ve noticed this skipping of items when performing delete actions (because my own process doesn’t even really handle it very well) and so I try to make my scripts idempotent so I can just run them again and pick up any records I missed the first time.

1 Like
#6

For the PDJS library, I discovered if you try to set the User-Agent header the browser will actually reject the entire request. So, we’ll just keep sending the library version through the URL as a query param.

We’ll also get it implemented in go-pagerduty.

1 Like
#7

Hey @simonfiddaman,

I hear you! The quick and simple solution is to use session.list_all or list(session.iter_all) to pre-fetch records ahead of time before running deletion, because deleting mid-iteration shifts the offset and causes records to get skipped.

1 Like
#8

Ooh, fancy! I’ll give it a shot! :slight_smile:

Cheers,

Simon